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^  The  Job  Skills  Education  Program  (JSEP)  is  designed  to  provide  soldiers 
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the  implementation  and  development  of  computer-based  instruction  '(Continued) 
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FOREWORD 


The  Job  Skills  Education  Program  (JSEP)  is  a  multi-phase  program  begun  in 
Fiscal  Year  1982,  and  designed  to  enhance  enlisted  career  potential  by 
improving  soldier  job  performance.  The  sponsor,  the  Education  Division, 

Office  of  the  Deputy  Chief  of  Staff  for  Personnel,  expects  JSEP  to  replace 
the  Army's  current  Basic  Skills  Education  Program  when  it  is  implemented. 

The  JSEP  program,  being  developed  by  Florida  State  University  (FSU)  will 
result  in  a  standardized  curriculum  for  soldiers  who  demonstrate  deficiencies 
in  the  knowledge  and  skills  required  to  successfully  learn  their  Military 
Occupational  Specialty  (MOS) . 

In  accordance  with  current  policy,  JSEP  will  be  an  on-duty  program.  It  will 
also  use  a  computer-based  management  system  to  facilitate  an  open  entry/open 
exit  approach.  At  present,  most  of  the  lessons  being  developed  will  be 
computer  delivered;  however,  the  plan  calls  for  using  existing  materials,  and 
incorporating  materials  developed  as  part  of  other  ARI  efforts,  whenever 
appropriate. 

A  unique  aspect  of  JSEP  is  that  it  builds  upon  a  very  detailed  front-end 
analysis  of  MOS  Baseline  Skills.  The  analysis  covered  tasks  performed  by 
soldiers  in  the  94  highest  density  MOSs,  in  addition  to  Common  Tasks  (the 
skills  that  all  soldiers,  regardless  of  their  MOS,  need  to  know).  Although  the 
Army  has  over  300  MOSs,  the  94  covered  in  the  analysis  represent  about  80%  of 
all  soldiers.  Perhaps  the  most  useful  product  developed  for  the  analysis  was 
a  taxonomy  listing  more  than  200  prerequisite  competencies. (P.C.)  for  these 
MOSs.  The  competencies  were  derived  from  detailed  reviews  of  Soldier  Manuals, 
and  from  extensive  interviews  with  subject-matter  experts  at  Army  schools. 

This  effort  produced  a  series  of  tests  intended  to  diagnose  deficiencies  in 
the  P.C.s.  Modified  versions  of  these  tests  will  be  used  in  JSEP. 

The  JSEP  program  will  include  a  front-end  learning  strategies  module 
designed  to  improve  soldier  skills  in  reading,  studying,  test  taking,  and 
problem  solving.  The  curriculum  will  consist  of  this  strategies-trainlng,  plus 
180  diagnostic  review  lessons,  and  120  skill  development  lessons,  which  are 
being  developed  for  the  PLATO  and  MicroTICCIT  computer  systems.  The  program  is 
being  tried  out  at  two  TRADOC  sites  and  two  F0RSC0M  sites,  prior  to  an  Army-wide 
phased  implementation. 
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EXECUTIVE  SUMMARY 


Requirement: 

The  solicitation  requires  that  a  semi-annual  technical  report  be  submitted 
for  Phase  II  to  the  Army  Research  Institute  (ARI). 

Procedure: 

This  report  constitutes  the  official  report  of  technical  and  devel omental 
processes  undertaken  in  the  first  six  months  of  Phase  II  of  the  Job  Skills 
Education  Program  (JSEP). 

Findings: 

Specific  technical  aspects  are  detailed  for: 
o  Project  Management 

o  Curriculum  Design 

o  Lesson  Design  and  Programming 

o  Data  Management  and  Student  Routing 

o  Testing  Procedures 

o  Tryouts  and  Field  Trials 

o  Implications  for  Demonstration  Year  and  Implementation 

Use  of  Findings: 

The  processes  reported  here  form  the  foundation  for  the  Tasks  for  Phase  II 
and  Phase  III. 
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Operational  Problem 


It  is  not  news  that  soldiers  must  be  trained  to  do  their  jobs.  They  must 
be  trained  so  that  each  Army  job  is  performed  competently— regardless  of 
differences  in  ability  and  background  in  newly  entering  soldiers.  To  accept 
less  would  cause  many  mission  elements  to  fail. 

Moreover,  many  Army  jobs  are  increasingly  dependent  upon  the  soldier's 
ability  to  use  high  technology  and  the  ability  to  learn  new  technology  as  it 
develops.  Soldiers,  therefore,  need  more  than  training.  They  need  enough 
education  to  be  able  to  learn  subsequent  jobs,  to  become  eligible  for 
promotion,  and  ultimately,  to  provide  leadership  for  tomorrow's  Army. 

The  Job  Skills  Education  Program  (JSEP)  is  designed  to  provide  soldiers 
with  job-related  basic  skills  instruction  that  is  prerequisite  to  learning 
their  skill  level  1  and  2  job  tasks  during  their  first  duty  assignment.  Based 
on  an  extensive  job  analysis  of  94  of  the  Military  Occupational  Specialties 
(MOS)  which  contain  the  largest  proportion  of  soldiers  and  tasks  contained  in 
the  Soldier's  Manual  of  Common  Tasks,  JSEP  provides  functional  basic  skills 
instruction  on  MOS  specific  requirements. 

As  it  is  conceptualized,  the  JSEr  curriculum  recognizes  cnac  tne  vast 
majority  of  soldiers  will  have  been  exposed  to  similar  basic  skills 
instruction  before  entering  the  Army.  Many  entering  soldiers,  however,  will 
not  have  learned  those  basic  skills  well  enough,  or  will  not  remember  what 
they  learned.  To  help  soldiers  learn  better  and  remember  more,  OSEP 
incorporates  straightforward  training  in  research-based  learning  strategies 
that  are  directly  aimed  at  improving  learning  and  retention. 
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Research  Objective 

The  goal  of  the  US  Army  Job  Skills  Education  Program  (OSEP)  is  to  design, 
develop,  and  test  a  job-related  computer  based  curriculum  for  possible  Army¬ 
wide  adoption.  The  JSEP  development  project  is  concerned  with  four  central 
elements: 

1.  The  computer  based  curriculum, 

2.  The  soldier  management  plan, 

3.  The  soldier  testing  program,  and 

4.  The  supporting  documents  and  reports. 

The  project  is  divided  into  three  phases.  Phase  I  undertook  several  tasks 
which  included  lesson  design,  planning  documents,  and  reports  that  recommend 
specific  courses  of  action.  Many  of  the  action  plans  spelled  out  in  the  Phase 
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I  reports  will  be  carried  out  in  Phases  II  and  Ill;  some  will  be  carried  out  in 
the  years  following  Phase  III. 

Phase  I  contains  the 'foil owing  tasks: 

Task  1.  Develop  a  conceptual  definition  for  design 
requirements  and  specifications 

Task  2.  Review  existing  basic  skills  curricula. 

Task  3.  Review  job  task  analysis  and  select  Military 

Occupation  Specialty  (M0S1  clustering  schema. 

Task  4.  Develop  JSEP  implementation  and  management 
plan. 

Task  5.  Adapt  or  develop  design  specifications 
including  instructi onal  specification, 
engineering  requi rements,  and  human  factors 
consideration. 

Task  6.  Develop  evaluation  standards. 

Task  7.  Develop  JSEP  evaluation  plan. 

Task  8.  Develop  cost  benefit  tradeoff  analysis. 

lasx  a.  uonauct  predictive  cost  ana  training 

effectiveness  analysis. 

Task  10.  Develop  standards  and  a  plan  for  the  civilian 
academic  community  to  award  a  high  school 
diploma  based  on  completion  of  OSEP. 

Task  11.  Phase  I  Report. 

Phase  II  requires  the  development  of  300  hours  of  lessons,  the  soldier 
management  plan,  and  the  preliminary  tryout.  Phase  II  includes  the  following 
tasks : 

Task  12.  Select  hardware;  where  appropriate  develop 

software;  develop  instructor  training  program 
and  courseware  for  300  hours  of  instruction. 

Task  13.  Conduct  preliminary  tryout. 

Task  14.  Phase  II  Report. 

Phase  III  provides  a  continuation  of  the  lesson  development  begun  in  Phase 

II  and  requires  execution  of  the  studies  and  evaluation  plans  developed  in 
Phase  i.  Phase  III  includes  the  following  tasks: 

Task  15.  Develop  software  and  courseware  for  an 
additional  120  hours  of  instruction. 


Task  16.  Conduct  JSEP  tryout  at  two  US  Army  Forces 
Command  (FORSCOM)  sites  and  two  US  Army 
Training  and  Doctrine  Command  (TRADOC)  sites. 

Task  17.  Conduct  evaluation  and  revision. 

Task  18.  Conduct  cost/training  effectiveness  analysis. 

Task  19.  Develop  technology  transfer  plan. 

Task  20.  Document  total  process  in  final  report. 

The  JSEP  Semi-annual  Technical  Report  is  a  collection  of  narratives  which 
cover  the  essential  issues  of  Phase  1  and  the  first  half  of  Phase  11  of  the 
contract.  These  issues  have  been  grouped  in  nine  topic  areas.  The  number  of 
issues  in  each  area  varies.  The  topic  areas  are: 


A.  Project  Management 

B.  Curriculum  Design 

C.  Lesson  Design  and  Programming 

D.  Data  Management  and  Student  Routing 

E.  Testing  Procedures 

C"  ^  o  *»  f  1  f  a  1 1’  rtn  /N  £  U  4  r*  Iw  r«n<w«l  £***».» 

G.  Tryouts  and  Field  Trials 

H.  Demonstration  Year 


I.  Implementation 

The  narrative  for  each  issue  will  describe  the  original  plan,  the  current  or 
implemented  plan,  and  the  reason  for  any  discrepancies  or  changes. 


A.  PROJECT  MANAGEMENT 
Computer  Appl ications 

Tracking  and  coordination  of  a  major  project  is  essential  to  timely 
completion.  Originally  the  critical  path  method  and  GANTT  charting  were 
proposed  as  the  main  management  tracking  technology.  While  these  methods  have 
proved  useful,  they  have  not  proved  sufficient.  They  do  not  provide  the 
detailed  information  necessary  to  take  corrective  action. 

Each  prerequisite  competency  must  be  tracked  through  its  individual  design 
stages  in  order  to  get  a  comprehensive  overview  of  the  project.  Also  budget 
projections  and  budget  status  are  a  critical  issue  with  a  large  staff. 

A  nationwide  search  was  conducted  for  computer  software  that  was  designed 
for  use  in  instructional  development  projects.  Failure  to  find  a  suitable 
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package  led  us  to  design  our  own  system  based  on  some  of  the  available 
business/management  software.  Our  preliminary  attempt  was  with  an  Apple 
computer  and  the  Vi  si -series.  We  used  the  Visidex  program  to  track  the 
individual  PC  specifications  produced  in  Phase  I,  but  it  was  cumbersome. 
VisiCalc  did  not  seem  to  have  enough  power  to  handle  our  budget  tracking 
procedures.  Our  second,  and  more  successful  efforts  were  made  with  an  IBM-PC 
and  the  Lotus  1-2-3  program.  With  the  addition  of  128K  memory,  Lotus  and  the 
IBM  gave  us  the  power  to  create  two  interacting  spreadsheets  which  encumbered 
and  projected  salaries  and  expenses.  Lotus  was  also  used  to  design  a  PC 
tracking  system.  Lotus  has  a  Macro  programming  function  which  allowed  us  to 
update  the  PC  tracking  system  and  to  print  out  various  versions  of  the  data  for 
monthly  reports  and  weekly  monitoring  with  a  minimum  of  reformatting. 

The  data  base  functions  of  Lotus  have  allowed  us  to  make  sequential 
entries  of  purchase  orders  and  charges  for  auxiliary  functions  and  then  sort 
the  lists  by  charge  designation,  which  greatly .facilitates  spreadsheet  updates. 

We  recommend  that  other  large  ISD  projects  explore  similar  options. 


B.  CURRICULUM  DESIGN 
Priority  for  Long  Lessons 
Setting  PC  Lesson  Development  Priorities 

Developing  detailed  instruction  for  all  PCs  would  result  in  far  more  than 
tne  approximately  nlu  nours  requireo  unoer  tms  contract.  *nereds  some  ?C> 
will  require  only  about  an  hour  to  complete,  some  will  take  far  more,  such  as 
36f,  Apply  common  rules  of  grammar,  which  will  probably  require  many  hours  of 
instruction  and  practice.  Short  lessons  will  be  developed  for  approximately 
170  testable  PCs.  (See  Table  7.)  In  an  attempt  to  establish  priorities  for 
long  lesson  development,  we  recommended  a  method  for  selecting  those  PCs  to  be 
developed  first. 

We  propose  that  the  factors  listed  below  be  taken  into  account  and  lessons 
tentatively  rated  high,  medium,  or  low  priority  (see  Appendix  A): 

o  How  many  MOS  require  the  PC? 

o  What  is  the  density  of  the  MOS  which  require  the  PC? 

o  Are  there  common  tasks  which  require  the  PC? 

o  Are  other  PCs  dependent  on  the  PC? 

o  Are  there  compatible  job-related  existing  materials  that  can 
be  integrated  into  JSEP  for  the  PC? 

o  How  did  the  TRADOC  schools  rate  the  PC? 

o  Is  the  PC  related  to  the  General  Technical  (GT)  portion  of  the 
ASVA3? 


o  F0RSC0M  Priority  rating. 


o  Is  It  appropriate  for  high  school  credential ing  program? 

Number  of  MOS.  In  order  to  answer  the  first  question,  the  RCA  MOS-PC 
Matrices  for  BSEP  I  and  BSEP  II  were  consolidated.  The  number  of  MOS  were 
counted  for  each  PC  (see  Table  8). 

For  example,  for  the  1  PC  series  except  li,  there  were  so  many  MOS  for 
each  PC  that  there  was  too  little  additional  information  to  be  gained  looking 
at  the  rest  of  the  questions.  This  PC  was  clearly  needed.  PC  la  through  lh 
were  tentatively  designated  high  priority.  PC  li  has  only  35  MOS  and  will  be 
classified  medium  priority. 

MOS  density.  Most  MOS  were  medium  density,  but  there  are  two  with  very 
high  density,  1 1C  with  about  12,000  soldiers  and  76Y  with  about  19,000. 
Therefore,  18a  is  a  medium  priority.  For  18b  and  18c,  all  of  the  MOS  have 
relatively  fewer  soldiers,  therefore  18b  and  18c  will  have  a  lower  priority. 

Common  tasks.  PC  series  la  -  li  (except  lg)  all  have  common  tasks.  PC  la 
through  lh  are  already  high  priority,  so  no  reconsideration  will  be  made.  PC 
li,  however,  had  previously  been  classified  medium  priority  and  would  probably 
be  reclassified  high  priority. 

Dependent  PCs.  Some  PCs  are  prerequisite  to  other  PCs.  If  a  PC  is 
prerequisite  to  a  high  priority  PC,  it  too,  must  become  a  high  priority.  PC 
lc,  Id,  and  le  are  prerequisite  to  If.  If  If  is  a  high  priority,  all  three, 
lc»  Id,  and  le,. should  become  high  priority. 

TRADOC  priority.  TRADOC  surveyed  the  proponent  service  schools  to  obtain 
priorities  for  all  JSEP  MOS.  The  schools  rated  each  PC  for  each  MOS  as: 

A  PC  already  taught  in  Advanced  Individual  Training  (AIT) 

1  Unanimous  agreement  priority  one  PC 

2  Mixed  responses  mostly  positive 

3  Mixed  responses  mostly  negative 

The  schools  further  rated  each  PC  as  to  whether  it  was  difficult  for  soldiers 
who  lacked  the  competency  to  learn  their  jobs.  An  A,  1,  or  high  difficulty 
rating  increases  the  FSU  priority  rating. 

GT  relationship.  If  a  PC  is  GT  related  it  automatically  becomes  high 
priorTTyl  A  PC  is  Gi  related  if  the  PC  matches  the  skills  tested  on  the  GT. 

Existing  materials.  A  PC  could  be  reduced  in  FSU  development  priority  if 
JSEP  compatible  existing  materials  were  available.  If  McFann-Gray's  (MGA) 
lesson  "Introduction  to  Numbers"  matched  the  indicator  statements  for  PC  la 
then  PC  la  would  be  taught  using  the  McFann*Gray  lesson.  The  MGA  materials  are 
primarily  drill  and  practice.  They  may  be  programmed  on  one  or  both  of  the 
computer  systems  used  to  deliver  JSEP. 

Level  of  difficulty.  RCA  recommended  developing  instruction  at  two 
difficulty  1  evel s.  Since  there  is  not  yet  any  data  on  what  would  be  difficult 
for  the  target  audience,  FSU  recommends  a  short  lesson  on  each  PC  of  the 
approximately  170  PCs  that  Educational  Testing  Service  (ETS)  declared  testable. 
Long  lessons  should  then  be  developed  for  higher  priority  testable  PCs. 


Raising  general  technical  aptitude  area  (GT)  scores.  One  goal  of  BSEP, 

BSEP  TH  and  the  JSEP  programs  since  'Task  1  oFTHase  1  is  to  increase  the 
chances  that  soldiers  taking  the  instruction  could  prepare  themselves  to  be 
eligible  for  reenlistment.  When  soldiers  are  eligible  for  reenlistment,  based 
on  their  test  scores,  the  Army  can  choose  from  a  much  larger  population  those 
to  recommend  for  reenlistment  based  on  their  job  performance  and  service 
records . 

Identifying  soldiers  who  need  to  raise  GT  scores  should  be  a.  part  of  the 

JSEP  management  system.  JSEP  soldiers  with  GT  scores  within  the  range  of 

eligibility  can  go  through  the  PCs  that  match  the  skills  tested  on  the  GT 
(arithmetic  reasoning,  paragraph  comprehension,  and  word  knowledge). 

The  proposed  long  lesson  priority  is  based  on  numbers  of  soldiers 

requiring  the  lesson,  the  RCA  analysis  of  ETS  -Test  results,  the  TRADOC  study  of 

difficulty  and  priority,  and  the  GT  relatedness  of  the  lesson. 

o  The  number  of  soldiers  criteria  gave  heavy  weight  to  the 
lessons  on  common  task  related  PCs.  High  density  MOS 
such  as  1 IB  or  lessons  required  by  more  than  40  MOS  added 
weight  to  non-common  task  PCs. 

o  RCAs  analysis  was  used  to  identify  those  lessons  on  which 
more  than  50*  of  soldiers  failed  the  ETS  test. 

o  The  TRADOC  study  collected  opinion  data  from  instructors 
and  for  each  PC  on  job  difficulty,  for  each  MOS  JSEP 
priority,  and  whether  the  PC  was  already  covered  in  the 
school . 

o  Lessons  related  to  GT  improvement  also  received  a  higher 
priority. 

FSU  recommends  a  decision  based  on  a  weighting  of  those  four  criteria  with 
the  following  factors  in  order  of  importance. 

1.  Common  task  PCs 

2.  ETS  Test  data 

3.  GT  improvement 

4.  High  number  of  soldiers  impacted 

Use  of  the  RCA  PC  -  MOS  Matrix.  The  RCA  PC-MOS  matrix  was  divided  into  2 
partsTTSEF  T^ncnTSTP  TlT^e  consolidated  the  data  and  came  up  with  a  JSEP 
PC-MOS  matrix.  We  planned  to  use  the  matrix  to  help  decide  lesson  development 
priorities. 

The  matrix  provides  an  excellent  quick  look  at  PC  series  to  see  how  many 
MOS  require  it.  Although  there  are  some  bugs  in  the  data  tape,  the  information 
has  been  very  useful.  We  recommend  a  similar  analysis  for  future  curriculum 
development  efforts.  A  summary  of  the  priorities  is  shown  in  Appendix  A. 


C.  LESSON  DESIGN  AND  PROGRAMING 
Lesson  Design 


One  of  the  initial  lesson  design  problems  faced  by  the  OSEP  staff  was 
deciding  upon  the  sequence  in  which  lessons  would  be  developed  in  the  JSEP 
curriculum.  Our  responsibility  is  to  produce  420  hours  of  instruction,  but 
what  would  be  the  priority  order  in  which  lessons  would  be  developed  for  the 
RCA  prerequisite  competencies? 

Various  proposals  for  setting  priorities  were  discussed  and  data  from 
various  sources  were  reviewed.  During  the  discussion  of  the  various  methods, 
the  observation  was  made  that,  for  any  particular  PC,  there  are  two  broad 
categories  of  soldiers  who,  at  first,  would  appear  to  require  instruction  on 
the  PC.  The  first  category  is  composed  of  those  soldiers  who  could  not 
correctly  answer  the  test  items  for  the  PC,  but  with  only  a  slight  review  would 
recall  prior  learning  and  be  able  to  perform  quite  well.  A  second  broad 
category  of  soldiers  includes  those  who  had  never  learned  the  skill  and 
probably  lacked  one  or  more  important  subordinate  skills. 

From  this  set  of  observations,  the  concept  of  short  and  long  lessons 
evolved.  In  general,  the  purpose  was  to  first  design  very  brief  lessons  (the 
short  lessons)  which  would  quickly  bring  up  to  speed  those  soldiers  who  had 
previously  learned  but  forgotten  the  skills.  Such  lessons  would  take  very 
little  time  of  those  who  had  never  learned  the  skill.  The  second  set  of 
lessons  (the  long  lessons)  were  to  serve  those  soldiers  who  required  in-depth 
instruction  in  order  to  produce  their  first  successful  performance  of  the 
skill. 


The  lessons  are  designed  as  follows: 

a.  Short  lessons.  Almost  all  of  the  PC's  can  be  considered  to  be 
(in  Gagne's  terminology)  intellectual  skills.  For  the  short  lessons, 
the  skill  is  described,  and  an  example  of  the  application  of  the  skill  is 
demonstrated.  Then  the  learner  is  provided  a  series  of  representative 
practice  and  feedback  items.  The  feedback  for  incorrect  responses  is 
sufficiently  detailed  to  clarify  misconceptions  and  bring  student 
performance  for  those  who  had  previously  mastered  the  skill,  to  an 
acceptable  level.  The  short  lessons  include  direct  instruction  on  only 
the  skill  described  in  the  PC,  and  not  on  any  of  the  subordinate  skills 
associated  with  the  PC. 

b.  Long  lessons.  The  long  lessons  are  intended  for  soldiers  who  didn't 
pass  the  test  for  the  short  lesson  on  the  PC  and  who  probably  lack  one  or 
more  of  the  relevant  subordinate  skills.  A  long  lesson  includes  both  a 
number  of  subordinate  skills  and  a  fuller  elaboration  of  the  events  of 
instruction  for  each  skill. 

To  develop  a  long  lesson,  the  designer  first  does  a  hierarchical 
analysis  to  identify  potential  subordinate  skills.  Then,  for  each  skill 
which  is  a  potential  problem  for  the  soldier,  a  complete  set  of 
instruction  is  provided  which  includes  motivational  pre-instructional 
activities,  a  complete  description  of  the  skill  and  an  appropriate  range 
of  examples.  An  ample  number  of  practice  and  feedback  problems  are 
provided,  along  with  alternate  remedial  instruction  where  applicable.  The 


’student  should  master  each  successive  subordinate  skill  before  returning 
to  instruction  on  the  PC  itself.  The  intent  is  to  provide  sufficiently 
extensive  instruction,  including  appropriate  practice  and  feedback,  that 
all  soldiers  will  be  able  to  perform  the  skill. 

The  testing  strategies  which  will  be  used  to  route  soldiers  into  and  out 
of  the  short  and  long  lessons  are  still  evolving.  The  basic  principle  is  that 
the  soldier  who  mostly  "knows”  the  skill  but  is  diagnosed  into  the  instruction 
will  exit  rapidly  after  completing  the  short  lesson,  while  the  soldier  who 
really  requires  the  instruction  will  receive  it,  in  depth,  in  the  long  lessons. 


Learner  Strategies 

During  Phase  I  we  conducted  an  extensive  review  of  the  literature  on 
various  learner  strategies  that  seemed  to  be  well-established  and  would  have  a 
potential  value  in  the  JSEP  curriculum.  Based  on  this  review  and  advice  from 
our  consultants,  Barbara  McCombs,  Donald  Dansereau,  and  Richard  Mayer,  we 
planned  a  six-module  learner  strategies  package. 

The  rationale  for  our  approach  to  strategy  training  is  outlined  in  a  paper 
by  Derry  entitled  Strategy  Training:  An  Incidental  Learning  Model  for  CAI  that 
was  presented  in  April  1984  at  the  annual  meeting  of  the. American  Educational 
Research  Association.  We  are  teaching  five  types  of  learner  strategies  in 
JSEP:  (1)  mood  management  techniques,  (2)  self-pacing  methods,  (3) 
comprehension  strategies,  (4)  memory  strategies,  and  (5)  problem-solving 
techniques.  Also,  we  plan  to  extend  the  problem-solving  techniques  to  a 
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Hardware  Choices 

There  were  many  hardware  options  initially  discussed  and  considered  for 
JSEP.  Ultimately,  the  hardware  options  decided  on  were  those  which  would 
provide  the  most  necessary  capabilities  across  the  largest  number  of  lessons. 
Hardware  options  adopted  for  MicroTICCIT  include  the  standard  MicroTICCIT 
workstation,  which  includes  light  pen,  detached  keyboard,  and  screen  printer. 
Hardware  options  adopted  for  PLATO  include  the  current  VIKING  terminals  with 
touch  panel  and  detached  keyboard,  screen  printer,  and  graphics  tablet. 

Hardware  options  which  were  considered  but  not  adopted  include  videotape, 
speech  generation,  sound  generation,  use  of  slides/film,  realtime  actual 
equipment  interface,  graphic  input,  and  use  of  optical  character/mark  sense 
readers. 

Some  hardware  options  might  still  be  desirable  for  inclusion  in  JSEP  at  a 
later  date.  For  example,  we  would  like  to  include  some  sounds  in  JSEP  lessons 
for  reinforcement  and  motivation.  It  would  be  desirable  to  include  interesting 
sounds  in  the  form  of  beeps,  buzzes,  and  sound  effects  (e.g.  airplanes 
crashing,  truck  starting,  bomb  exploding,  etc.).  At  the  present  time  these 
capabilities  are  not  practical  for  either  the  PLATO  or  MicroTICCIT  systems. 
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Standard  on  Production  of  Lessons  on  MicroTICCIT 

During  most  of  Phase  11  production,  we  have  used  the  TICCIT  Authoring 
Language  (TAL).  We  made  the  decision  to  use  TAL  because  it  gave  us  the 
following  capabilities. 

1.  Freedom  to  write  Interactive  Rules. 

2.  Direct  conversion  to  ADAPT.  When  ADAPT  is  released  we  will 
convert  all  existing  TAL  programs  and  complete  all  production 
in  the  new  language. 

3.  Flexibility  in  lesson  content  and  structure. 

The  alternative  to  TAL,  the  Authoring  Procedure  for  TICCIT  (APT),  is  a 
highly  structured  authoring  environment  which  presently  prevents  interactive, 
rules,  direct  conversion  with  ADAPT  programming  code,  and  multiple  lesson 
formats. 


ADAPT  is  designed  to  be  The  language  for  MicroTICCIT,  and  will  be  released 
shortly.  We  will  use  ADAPT  because  it  combines  the  efficiency  in  production  of 
APT,  derived  from  its  templated-formats,  with  the  power  and  flexibility  of  TAL. 
ADAPT  also  has  new  features  that  run  on  MicroTICCIT  such  as  color  palette,  and 
video. 


Because  our  current  production  system  does  not  have  any  MicroTICCIT 
terminals  we  have  not  used  ADAPT's  new  features.  However,  shortly  after  we 
cb ta i r.  A..AP 7 ,  wc  w *  7 »  ct La  ■  r»  severa.  K •  c re . * w . »  r m \ r»a *  s ,  sc  c. 
system  will  match  the  delivery  system  in  all  important  respects. 
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D.  DATA  MANAGEMENT  AND  STUDENT  ROUTING 
Comments  About  the  JSEP  Data  Management  System  ( DMS ) 

As  of  this  writing,  the  DMS  exists  as  an  emergent  set  of  assumptions  and 
interrelated  design  decisions.  During  the  past  year,  our  plans  for  the  DMS 
have  had  to  be  modified  to  accommodate  the  additional  goal  of  GT  improvement. 

We  have  also  recently  re-examined  our  selection  and  routing  criteria  because  of 
the  poor  match  between  the  RCA  Locator  and  Diagnostic  tests  and  our  PC 
objectives.  No  final  decisions  have  been  made  regarding  the  new  criteria. 

The  DMS  will  be  the  "control  center"  for  USEP  and  will  collect  demographic 
data  and  lessons  performance  data  and  will  use  those  data  to  provide  routing 
options  to  soldiers.  The  system  will  be  flexible  enough  to  account  for  the 
needs  of  different  sizes  of  Army  bases.  At  large  bases,  the  site  managers  and 
instructor  will  be  allowed  control  over  the  DMS  at  specific  input/output 
points.  At  small  bases,  the  DMS  will  be  able  to  function  without  any 
instructor  involvement. 
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Regardless  of  the  special  conditions  at  each  site,  we  recommend  that  the 
DMS  operate  roughly  as  follows: 

1.  The  DMS  will  collect  demographic  data  about  soldiers.  This 
will  include  scores  on  the  GT. 

2.  The  system  will  perform  a  front-end  analysis  to  determine 
each  soldiers'  recommended  program  of  stutjy.  All  data  to  be 
used  in  this  analysis  has  not  yet  been  identified. 

3.  Soldiers  will  obtain  a  copy  of  their  program  of  study  and 
discuss  it  with  their  instructor. 

4.  They  will  then  proceed  through  their  program  of  study, 
controlling  their  own  pace  and  sequence  of  lessons.  The 
instructor  will  be  involved  as  necessary.  Soldiers  will 
report  all  completions  to  the  instructor  to  update  an  off¬ 
line  record  of  their  progress. 

5.  As  soldiers  proceed  through  their  program  of  study,  the  DMS 
will  collect  and  make  available  to  instructors  performance 
data  ranging  from  test  scores  to  time  in  each  lesson. 


Potential  Routing  Mechanisms 

Four  independent  measures  were  explored  as  potential  screening  mechanisms 
to  neip  soldiers  identity  lessons  in  wmcn  tney  nave  a  mgn  nxennooo  ot 
requiring  instruction.  (See  correlation  matrix  on  Table  1.)  These  were  the: 

(1)  JSEP  Achievement  Test  used  as  a  pretest, 

(2)  GT  scores, 

(3)  RCA  Locator  Verbal,  and 

(4)  RCA  Locator  Math. 

These  measures  were  correlated  with  pretest  scores  of  the  seventeen  lessons 
used  in  the  tryout.  It  was  hoped  that  one  of  these  measures  would  demonstrate 
consistent  positive  significant  correlations  across  a  majority  of  the  lessons. 
If  this  had  been  the  case,  then  scores  on  the  screening  measure  could 
predict  whether  a  soldier  would  likely  require  instruction  in  the  lessons.  The 
results  of  the  correlational  analyses  are  presented  in  Table  2. 
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Table  2 


Correlations  of  Potential  Predictors  of  Lessons  Pretest  Scores 


Lesson  Pretests  OSEP  Achievement  GT  Locator  Locator 

Pretest  Verbal  Math 


J 

1 

1. 

30b  Flowchart 
(n  *  39) 

1 

2. 

32a  Blocks  in  a  Form 
(n  =  39) 

1 

3. 

32b  Forms 
(n  =  37) 

■ 

4. 

32c  Forms 
(n  =  17) 

t 

5. 

32d  Forms 
(n  =  28) 

ft 

6. 

32e  Forms 
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Clauses 
(n  -  39) 

3.  36c  Capitalization 

(n  =  29) 

?.  40a  Hazards 

(n  =  37) 

10.  40b  Preventive  Measures 
(n  =  39) 

11.  40c  Emergencies 
(n  *  39) 

12.  Olg  Rounding  Numbers 
(n  =  39) 

13.  5f  Gages 
(n  =  17) 

14.  26b  Technical  Vocabulary 
(n  *  39) 

15.  28a  Tables  and  Charts 


Table  2  Continued 


Lesson  Pretests 

JSEP  Achievement 
Pretest 

GT 

Locator 

Verbal 

Locator 

Math 

16.  28b  Tables  and  Charts 

(n  =  18) 

-.05 

.33 

.14 

-.32 

17.  28c  Tables  and  Charts 

(n  =  17) 

-.25 

.15 

.02 

.08 

Mean  Correlation 
across  all  lessons 

-.08 

.08 

.02 

-.05 

*  p  <  .05 


The  results  of  the  correlational  analysis  indicate  the  mean  correlation 
between  the  potential  routing  measures  and  the  lessons  were:  JSEP  Achievement 
Pretest,  M  =  -.08;  GT,  M  =  .08;  Locator  Verbal,  M  =  .02  and  Locator  Math,  M  = 
-.05.  The  correlations  between  predictor  measures  and  lesson  pretest  measures 
virtually  clustered  around  zero.  We  concluded  that  no  single  score  from  any  of 
these  messure?  is  likely  to  consistently  predict  performance  across  lessor, 
pretests.  The  potential  use  of  any  of  these  as  screening  devices  appears 
remote. 

This  lack  of  relationship  between  general  measures  and  the  more  specific 
lesson  pretests  could  be  attributed  to  several  sources: 

1.  The  unreliability  of  the  10-15  item  lesson  tests  could 
attenuate  the  correlations  to  levels  below  significance 
in  most  lessons. 

2.  Since  the  distributions  of  scores  of  the  pretests  for  many 
of  the  lessons  were  so  highly  skewed  toward  the  positive 
side,  correlations  were  consequently  suppressed  due  to 
non-linear  relationships  between  variables. 

3.  Truncated  ranges  in  the  general  ability  measures  would 
reduce  the  potential  for  correlation  with  lesson  pretests. 

GT  scores  80-99  represent  the  3rd  quantile  in  the 
distribution  of  measures  of  general  intelligence. 

4.  One  would  not  expect  the  JSEP  test,  comprised  of  3  items 
randomly  selected  from  11  discrete  knowledge  units,  to 
predict  mastery  of  a  single  knowledge  unit.  There  is  only 
1/11  of  the  achievement  test  domain  in  common  with  any 
single  pretest. 


’■’The  implications  of  these  findings  regarding  the  use  of  a  single  routing 
mechanism  may  include  the  following: 

a.  Administer  these  general  prediction  measures  in  the  Full  Scale  tryout 
to  validate  the  above  results  on  a  wider  array  of  lessons,  but  do  not 
use  them  as  either  screening  or  routing  mechanisms  in  the  tryout. 

b.  Conduct  an  item  analysis  on  the  lesson  posttests  using  another  sample 
of  soldiers  to  possibly  improve  the  tests  items  and  hence  their 
reliabilities. 

c.  Explore  the  possibility  of  clustering  lessons  on  the  basis  of 
underlying  cognitive  ability  factors  and  then  using  the  general 
measures  to  predict  to  these  ability  factors.  These  factors  could  be 
identified  possibly  through  factor  analytic  procedures  or  other 
computer  sorting  methodologies.  Again,  recommendations  pertaining  to 
the  use  of  such  procedures  in  the  implementation  phase  would  be  made 
from  additional  data  gathered  from  the  Full  Scale  Tryout. 


Table  3 


General  Performance  Measures 


Measure 


Pretest 


Posttest 


Effect  size 


Mean 

SD 

Mean 

SD 

1. 

JSEP  Achievement  Test 
(39  possible) 

28.6 

3.4 

33.3 

2.6 

1.38 

2. 

RCA  Performance  Test 
(79  possible) 

48.7 

11.8 

49.1 

13.5 

.03 

3. 

CLOZE 

(35  possible) 

— 

— 

14.6 

6.0 

— 

4. 

GT 

91.2 

4.9 

_  _ 

—  _ 

_  —  _ 

The  JSEP  Achievement  Test 

The  JSEP  Achievement  Test  was  developed  by  selecting  at  random  three  items 
or  tasks  from  the  posttests  of  11  lessons.  Further  development  of  the  test 
would  render  the  instrument  a  sensitive,  reliable,  and  valid  proximal  measure 
of  \}$ED 


RCA  Performance  Test 

The  items  selected  for  the  performance  test  were  drawn  from  the  RCA 
Criterion  Test.  These  items  were  generally  consistent  with  the  content  of  the 
JbtP  lessons  out  not  always  exactly  so.  In  many  instances  the  complexity  of 
the  stimuli  was  greater  in  the  Performance  Test  than  in  the  items  of  the  lesson 
posttests.  For  example,  the  number  of  process/decision  steps  in  the  lesson  on 
flowcharting  was  3-5  (as  many  as  could  be  presented  on  a  display  screen) 
whereas  the  Performance  Test  contained  15-20  such  steps.  Items  with  electronic 
circuitry  and  rounding  numbers  tasks  were  also  more  complex  than  material 
presented  in  the  computer-based  lessons.  Thus,  there  was  an  increment  not  only 
in  difficulty  from  the  nature  of  the  assessment  task  (recognition  to  recall) 
but  also  in  terms  of  the  complexity  of  the  stimulus  function. 

Therefore,  in  order  to  construct  a  more  sensitive  and  valid  recall 
measure,  we  should  begin  from  our  own  JSEP  Achievement  Test  as  a  point  of 
departure.  From  the  pool  of  JSEP  Achievement  Test  items  we  could  perform  the 
following: 

1.  Select  one  JSEP  Test  item  from  each  lesson  comprising 

the  set  of  GT  prep  lessons  and  the  set  of  KOS  common  task 
lessons.  Two  Performance  Tests  would  then  be  created,  one  for 
MOS  Common  Tasks  and  one  for  the  GT  preparatory  course. 
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2.  The  selection  would  be  based  on  such  considerations  as 

resemblance  of  the  stimulus  context  to  on-the-job  situations 
and  the  ease  and  objectivity  of  scoring  the  response. 


CLOZE  Test 

The  CLOZE  procedure  was  used  as  a  test  because  it  is  intended  to  measure 
reading  comprehension  and  in  this  case  to  assess  the  ability  to  read  the 
soldier's  field  manual.  The  CLOZE  Test  used  in  the  preliminary  tryout  was 
created  by  selecting  a  300-word  .paragraph  with  accompanying  diagrams  from  the 
Manual  of  Common  Tasks.  Every  eighth  word  was  omitted  from  the  paragraph.  In 
all,  thirty-five  words  were  omitted  from  the  paragraph.  The  content  of  the 
passage  concerned  procedures  for  the  proper  aiming  of  a  rifle. 

The  results  of  the  trial  test  of  the  CLOZE,  procedure  indicated  that  the 
group  of  soldiers  attained  a  mean  of  14.8  with  a  standard  deviation  of  6.0.  The 
scores  were  normally  distributed  about  the  mean.  Only  one  soldier  was  unable 
to  begin  the  task. 

While  the  CLOZE  procedure  is  designed  to  measure  a  general  ability  to 
comprehend  written  prose,  it  is  questionable  whether  it  could  or  would  be  an 
appropriate  outcome  measure  for  the  JSEP  curriculum.  The  JSEP  curriculum  is 
structured  in  terms  of  many  narrowly  defined  skill  and  content  units  that  can 
be  learned  in  20  minutes  to  1  hour.  The  question  is  raised  as  to  whether 
learning  from  specific  learning  units  will  be  integrated  and  transfer  to  more 
general  abilities,  such  as  comprehending  a  paragraph  of  prose  in  a  field 
ffidnud  •  • 

Secondly,  the  nature  of  the  passage  that  was  selected  was  very  different 
from  most  of  the  written  material  typically  found  in  the  FM's.  The  more 
familiar  prose  in  the  manual  is  in  the  form  of  stepwise  lists  of  procedures, 
diagrams,  and  flowcharts  to  conduct  a  given  task. 

For  these  reasons,  it  is  probably  unwise  to  use  the  CLOZE  procedure  as  a 
JSEP  outcome  measure  to  assess  the  ability  to  comprehend  FKs. 


The  GT 

One  of  the  purposes  of  JSEP  is  to  assist  soldiers  in  achieving  a  better 
performance  on  the  GT  (General  Technical  Subtest  of  the  ASVAB).  Soldiers  were 
selected  for  the  Tryout  on  the  basis  of  their  GT  scores--that  is,  they  earned 
GT  scores  between  80  and  99.  Thus  the  GT  score  distribution  of  M  =  91.2,  SD  = 
4.9  differed  from  the  Army  wide  distribution  M  =  100,  SD  =  20.0.  Given  the  GT 
score  distribution  of  the  expected  JSEP  population,  in  subsequent  field  trials 
we  will  likely  observe  effects  of  JSEP  instruction  not  only  in  terms  of  changes 
in  the  mean,  but  also  in  terms  of  changes  in  the  distribution  of  scores  about 
the  mean. 


Use  of  Job  Performance  Measures  to  Test  Effect  of  JSEP 


Use  of  job  performance  measures  was  originally  proposed  for  JSEP 
evaluation;  however,  there  are  certain  technical  difficulties. 

JSEP  could  be  evaluated  in  a  longitudinal  study  provided  that  there  were 
enough  implementation  sites  to  have  sufficient  soldiers  to  study.  A 
longitudinal  study  of  this  magnitude  could  be  implemented  by  Education 
Division,  ODCSPER,  TRADOC,  or  FORSCOM.  It  would  take  three  to  four  years. 
SQT's  and  other  measures,  as  yet  not  fully  identified,  could  be  used  as  the 
criterion  variable  for  correlations. 


G.  TRYOUTS  AND  FIELD  TRIALS 

The  JSEP  Field  Trials  were  conducted  at  Fort  Rucker,  5-18  Kay  1984. 


The  Population 

By  directive,  we  requested  that  soldiers  be  included  in  the  tryout 
population  who  met  two  conditions: 

(1)  they  had  earned  GT  (ASYA8  General  -Technical )  scores  between  80  and  99, 

and 

(2)  for  the  most  part  they  had  been  recently  stationed  at  Fort  Rucker. 

The  MOS  represented  in  the  tryout  were  67  (N,  V,  Y),  76  (N,  W,  Y),  91B, 
and  94B.  Background  data  in  terms  of  age,  rank,  sex  race,  time  in  the  Army  and 
time  on  the  post  are  presented  in  Table  4  below.  Generally  speaking,  the  42 
soldiers  participating  in  the  tryout  had  been  in  the  Army  less  than  15  months, 
at  Fort  Rucker  less  than  8  1/2  months  and  in  their  respective  MOS  8  months. 
Sixty-seven  percent  (67%)  had  ranks  E3  or  below.  In  order  to  achieve  the 
required  number  of  participants  for  the  tryout,  the  length  of  time  soldiers  had 
been  at  Fort  Rucker  had  to  be  extended  beyond  the  desired  3  month  limit.  One 
soldier  was  a  master  sergeant  who  had  been  in  the  Army  more  than  9  years. 
Nevertheless,  the  GT  parameters  were  not  extended  in  either  direction. 

Over  half  (55%)  of  the  population  indicated  they  will  seek  reenlistment. 

If  this  is  the  case,  they  will  most  likely  require  some  assistance  to  raise 
their  GT  scores  in  order  to  qualify  for  reenlistment.  Thirty-eight  percent 
(38%)  indicated  they  were  not  sure  whether  they  wanted  to  reenlist.  Perhaps 
some  of  the  doubt  in  this  group  could  be  attributed  to  lack  of  confidence  in 
their  ability  to  meet  desired  minimum  standards  on  the  GT.  Some  had  already 
undertaken  study  to  improve  basic  skills:  24?  had  taken  a  BSEP  reading  course, 
10%  a  grammar  course,  and  29?  a  mathematics  course.  Ninety-three  percent  had 
attained  a  high  school  diploma. 

On  the  basis  of  background  characteristics,  Army  career  goals  and 
educational  experiences,  it  is  highly  likely  that  most  in  the  Fort  Rucker 
sample  represent  soldiers  Army  wide  who  would  be  interested  in  taking  JSEP 
lessons  to  improve  basic  skills,  particularly  for  reenlistment.  Therefore,  we 
conclude  that  most  of  the  soldiers  in  the  Fort  Rucker  tryout  were 
representati ve  of  potential  JSEP  populations. 


Table  4 


Sample  Background  Characteristics  of  Fort  Rucker  Tryout  (n=42) 

Dimension 

Frequency 

Dimension 

Frequency 

Age 

mean  =  22.4 
median  =  20.7 

SO  =  4.4 

range  =  40-18 

Time  in  present 
job  (months) 

mean  =  14.5 
median  =  8.0 

SD  =  18.8 

range  =  2-84 

Rank 

El  (  1) 

E2  (14) 

E3  (13) 

E4  (  9) 

E5+  (  5) 

Time  on  po.s.t 
(months ) 

mean  =  11.7 
median  =  8.5 

SD  =  11.2 

range  =  l-42mo 

Sex 

male  34 

female  8 

High  school 
diploma 

yes  =  39(92.9%) 
no  =  3(7.1%) 

Race 

white  19 

black  17 

hispanic  6 

Highest  grade 
compl eted 

9th  1 

10  th  1 

11th  1 

12th  38 

13th  1 

MOS 

67N,  V,  Y  (11) 

76N,  W,  Y  (16) 

9 IB  (  8) 

94B  (  7 ) 

Seek  reenlistment? 

don 

yes  *  23(55%) 
't  know  =  16(38%) 
no  =  3(7%) 

Time  in 

Army 

(months) 

mean  =  25.7 
median  =  15.5 

SD  =  25.2 

range  =  5-99+ 

Any  postsecondary 
education? 

yes  =  4 
no  *  38 

Taken  a  BSEP 

reading 

course? 

yes  =  10(24%) 

no  =  32 

Had  a  previous 
computer  course 

yes  =  7 

no  =  35 

Taken  a  BSEP 

grammar 

course? 

yes  =  4(10%) 

no  =  38 

Had  a  previous  Audio- 
visually  oriented 
course 

yes  =  11 

no  =31 

Taken  a  BSEP 
mathematics 

yes  =  12(29%) 

no  =  30 

=  3(70%) 

=  39 


Taken  an  ESL  yes 
course  no 


Effectiveness  of  Lessons 


f 


The  Fort  Rucker  tryout  provided  an  opportunity  to  examine  the  quality  of 
lessons  on  an  actual  client  population.  Until  the  preliminary  tryout,  the 
lessons  had  undergone  only  internal  reviews  and  some  one-on-one  testing. 

Because  we  were  unable  to  conduct  extensive  formative  development  of  the 
lessons,  important  questions  pervaded  early  development  such  as:  Are  the 
lessons  too  difficult  or  too  easy?  Will  they  sustain  sufficient  interest  and 
attention?  Are  the  screen  displays  attractive?  How  much  can  soldiers  learn  in 
a  short  20  minute  recall  lesson?  How  much  can  soldiers  learn  in  a  longer  one- 
hour  lesson?  The  tryout  offered  an  initial  examination  of  such  issues. 

The  difference  between  soldiers'  pretest  and  posttest  scores  on  the  17 
lessons  are  presented  in  Table  5.  Across  all  lessons  there  was  an  average 
median  gain  score  of  8.0%.  There  were  two  lessons  in  which  there  was  a  20% 
gain  or  higher,  five  in  which  there  was  a  10%  gain  or  higher,  five  in  which 
there  was  a  5%  gain  or  less.  The  long  lessons  (1  hour  of  CAI)  produced  an 
average  of  14.1%  gain  while  the  short  lessons  (15  minutes  including  the  test) 
produced  a  6.1%  average  median  gain. 


Table  5 


Cumulative  Percent  of  Soldiers  Scoring  at  Percent  Level  or  Higher 

on  Pretest  and  Posttest 


Lesson 
•  * 

50 

60 

70 

80 

90 

Median 

Or 

Median 

* 

Correct 

Gain 

Long  lessons 

I.  01 g  Rounding 

Pretest 

74 

49 

31 

26 

8 

60.3 

+9.7 

Numbers 
( n=40) 

Posttest 

80 

62 

52 

35 

25 

71.0 

2.  5f  Gages 

Pretest 

100 

100 

88 

65 

47 

83.3 

+11.7 

(n-12) 

Posttest 

100 

100 

92 

83 

75 

95.0 

3.  26b  Technical 

Pretest 

46 

22 

8 

5 

3 

47.8 

+31.5 

Vocabulary 

Posttest 

95 

81 

65 

49 

30 

79.3 

(n*37) 

4.  36c 

Pretest 

79 

72 

59 

31 

3 

71.3 

+  3.6 

Capitalization 

(n*23) 

Posttest 

83 

70 

61 

30 

9 

74.9 

Short  lessons 

FI  ?8a  lables 

Pretest 

87 

80 

80 

73 

47 

88.7 

+  9.3 

and  Charts 
(n-12) 

Posttest 

92 

92 

92 

75 

67 

97.0 

6.  28b  Tables 

Pretest 

78 

72 

61 

56 

22 

82.7 

-  1.2 

and  Charts 

Posttest 

88 

88 

71 

65 

18 

81.5 

(n»17) 


Table  5  Continued 


7.  28c  Tables 

and  Charts 
^*14) 

Pretest 

Posttest 

94 

86 

65 

86 

47 

79 

18 

57 

18 

21 

67.0 

86.0 

+19.0 

8.  30b  Flow 

Pretest 

94 

79 

64 

46 

5 

78.4 

+  9.0 

Chart 
( n*40) 

Posttest 

90 

87 

82 

67 

32 

87.4 

9.  32a  Locate 

Pretest 

100 

100 

95 

92 

90 

99.8 

+  0.1 

Blocks  in  a 

Posttest 

100 

100 

100 

100 

95 

99.9 

Form  (n=39) 

10.  32b  Forms  B 

Pretest 

100 

100 

'97 

89 

84 

97.6 

+  1.2 

(n-31) 

Posttest 

100 

100 

100 

100 

93 

98.8 

11.  32c  Forms  C 

Pretest 

88 

88 

88 

82 

76 

97.8 

-  0.9 

(n-13) 

Posttest 

92 

85 

77 

77 

77 

97.9 

12.  32d  Forms  D 

Pretest 

86 

79 

71 

54 

11 

75.8 

+  4.7 

(n-28) 

Posttest 

100 

96 

82 

71 

32 

80.5 

13.  32e  Forms  E 

Pretest 

100 

93 

90 

69 

52 

88.8 

-  5.8 

(n=29) 

Posttest 

100 

100 

86 

59 

41 

83.0 

14.  34a  Major  and 

Pretest 

72 

51 

28 

20 

8 

55.6 

+17.7 

Subordinate 

Posttest 

82 

74 

69 

46 

38 

73.3 

Clauses 

(n=39) 

15.  40a  Hazards 

Pretest 

100 

100 

100 

100 

92 

99.5 

+  0.2 

(n-30) 

Posttest 

100 

100 

100 

100 

93 

99.7 

16.  40b  Preventive 

Pretest 

100 

100 

100 

85 

64 

89.6 

+  2.0 

Measures 

Posttest 

100 

100 

95 

90 

72 

91.6 

(n=39) 

17.  40c  Pretest  56  51  28  15  0  55.6  +24.0 

Emergencies  Posttest  95  92  73  65  32  79.6 

( n=37 ) 


The  conclusions  pertaining  to  the  lesson  performances  are  that  the  long 
lessons  appear  to  help  individuals  increase  mastery  of  lesson  objectives  but 
that  the  effect  of  short  lessons  as  "refreshers"  varied  considerably  across 
lessons.  The  largest  gains  were  in  lessons  that  had  low  pretest  scores  and 
dealt  with  concrete  verbal  or  figurative  learning  (e.g.,  Technical  Vocabu i ary , 
Major  and  Subordinate  Clauses,  Emergencies).  The  more  abstract  the  concepts  or 
the  more  a  lesson  stressed  the  application  of  rules,  the  less  was  the  gain 
(Capitalization  and  Rounding).  Over  half  of  the  lessons  (nine  of  seventeen) 
had  median  pretest  scores  exceeding  80%  correct.  Thus,  there  was  a  pronounced 


celling  effect  on  the  amount  of  potential  gain  the  lessons  could  demonstrate. 
Finally,  while  the  recognition  mastery  of  the  lessons  was  demonstrated  and 
sustained  over  several  days,  the  amount  of  recall  mastery  was  negligible.  The 
validity  of  the  recall  test  however  may  be  suspect,  because  the  test  items  were 
more  intricate  and  complex  than  the  test  items  in  the  lesson  tests. 

The  third  aspect  of  the  evaluation  concerned  the  attitudes  soldiers  had 
toward  the  quality  of  the  lesson,  and  the  conditions  under  which  JSEP  would 
best  be  administered.  The  responses  of  soldiers  are  presented  in  Table  6. 


Table  6 

Attitudes  Toward  JSEP 


1  =  strongly  agree 

2  =  agree 

3  =  uncertain 

4  =  disagree 

5  =  strongly  disagree 


Conditions  for  Learning 


»  v  .  >*. 


ww  W  J  L  VI  i  C  WlH|/WbCi 


2.  Using  the  computer  for  four  hours  at  a  time  was 
too  long. 

3.  I  think  written  assignments  should  be  used  along 
with  the  computer. 

Soldier  Impact 


4.  I  think  the  lessons  will  help  me  to  read  and 
understand  the  publications  I  use. 

5.  The  skills  I've  learned  in  JSEP  will  help  me  to 
advance  to  a  higher  grade/rank  in  the  Army. 

Army  Related  Concerns 


I  think  the  instructor  in  the  Education  Center 
should  teach  the  lessons  instead  of  the  computer. 

I  would  be  willing  to  take  more  JSEP  lessons  on  a 
computer  if  they  were  offered  during  on-duty  hours. 

I  would  be  willing  to  take  more  JSEP  lessons  offered 
on  a  computer  if  they  were  offered  during  off-duty 


Table  6,  Continued 


9.  I  think  my  unit  commander  would  be  willing  to  release  2.31  .87 

me  from  duty  to  take  JSEP  lessons. 

10.  I  think  JSEP  should  be  Included  as  an  educational  1.62  .66 

program  offered  by  the  Education  Center. 


We  conclude  that  the  soldiers  enjoyed  the  lessons  and  felt  they  learned 
from  them.  However,  the  majority  of  soldiers  indicated  they  would  not  elect  to 
take  the  lessons  off-duty.  However,  there  was  considerable  variability  among 
participants.  The  soldiers  strongly  preferred  CBI  to  regular  instructor 
delivered  lessons. 


Student  Response  to  Screen  Formats 

The  purpose  of  the  JSEP  General  Questionnaire  was  to  assess  soldier 
attitudes  about  the  JSEP  program  as  a  whole  and  how  JSEP  might  suit  their  own 
and  the  Army's  goals.  The  subjects  were  42  soldiers  participating  in  the  Fort 
Rucker  preliminary  tryout  with  GT  scores  between  80  and.  99.  They  were  exposed 
to  a  one-week,  four  hours  per  day  simulation  of  the  JSEP  program.  The 
instrument  was  administered  on  the  last  day  of  their  assigned  duty  time  and  the 
soldiers  were  informed  that  they  were  not  required  to  record  their  names  on  the 
survey. 


Results 

Question  1. 


It  was  easy  for  me  to  learn  to  use  the  computer. 


Strongly 

Agree 

Agree 

Uncertain 

Di sagree 

Strongly 
Di sagree 

45% 

50% 

2% 

0% 

2% 

Comment:  The  brief  orientation  procedures  used  to  orient  individuals  to  the 
computer  appeared  to  be  fully  adequate  for  95*  of  the  subjects.  Appropriate 
sign-on  procedures  were  demonstrated  by  the  proctors,  and  the  existing  self- 
instructional  computer  programs  on  MicroTICCIT  and  PLATO  directed  them  from 
there.  Minimal  individual  coaching  and  encouragement  was  given  by  the 
proctors. 


••Question  2. 


Using  the  computer  for  four  hours  at  a  time  was  too  long. 


Strongly 

Agree 

Agree 

Uncertain 

Disagree 

Strongly 

Disagree 

0% 

20% 

12% 

46% 

22% 

Comment:  Sixty-eight  percent  of  the  soldiers  disagreed  with  the  statement  and 
only  19%  indicated  four  hours  is  too  long.  Thus,  four  hours  was  not  considered 
to  be  too  long  a  time  in  which  to  receive  computer-based  instruction. 


Question  _3- 

I  think  the  instructor  in  the  educational  center  should  teach  the 
lessons  instead  of  the  computer. 


Strongly 

Agree 

Agree 

Uncertain 

Di sagree 

Strongly 

Disagree 

0% 

0% 

n 

57% 

36% 

Comment:  93“  of  the  soldie-s  disagreed  with  this  statement. 
Question  4.  (Bogus  Question). 


Question  5. 

I  think  computer  lessons  should  be  used  only  to  practice  what  we 
learn  from  an  instructor. 

Strongly  Strongly 

Agree  Agree  Uncertain  Disagree  Disagree 

0%  17%  14%  45%  24% 

Comment:  This  was  a  question  to  help  determine  how  a  computer  should  be  used 
in  an  instructional  context.  More  specif ically,~wFether  it  should  be 
restricted  to  use  as  an  adjunct  to  conventional  instruction.  Most  soldiers 
(69%)  disagreed  that  it  should  be  used  exclusively  as  a  complementary  drill  and 
practice  medium. 


Question  6. 


I  think  written  assignments  should  be  used  along  with  the 
computer. 


Strongly 

Agree 


Strongly 

Agree  Uncertain  Disagree  Disagree 


10%  38%  14% 


33%  5% 


Comment:  A  question  was  raised  by  one  of  the  trial  observers  that  some 
soldiers  said  they  would  like  print  materials  to  go  along  with  the  lessons. 
Forty-eight  percent  of  the  JSEP  subjects  agreed  with  the  statement  while  38% 
disagreed.  Thus  there  is  considerable  ambivalence  regarding  the  utility  of 
such  material.  Another  way  of  looking  at  the  issue  is  that  only  15%  felt 
strongly  one  way  or  the  other. 


Question  7_. 

The  lesson  about  mood  management  helped  me  control  my  mood  during 
JSEP  lessons. 


Strongly 

Strongly 

Agree 

Agree 

Uncertain  Disagree  Disagree 

’-’O' 

—  K 

“Ji  o  ^  ^  O'  *  rso> 

C,  W  A  «  A  W  *. 

Comment:  The  soldiers  appeared  to  be  mainly  uncertain  or  not  strongly 
influenced  by  the  mood  management  experience.  The  extent  of  ambivalence  or 
uncertainty  might  have  been  due  to  several  factors:  (1)  the  fact  that  the 
lessons  were  administered  in  paper  format,  a  medium  soldiers  did  not  seem  to 
prefer;  (2)  the  lack  of  perceived  relationship  between  the  mood  management 
lesson  and  subsequent  JSEP  lessons;  and  (3)  the  absence,  on  the  part  of  a 
segment  of  soldiers,  of  a  perceived  need  for  such  instruction. 


Question  8 . 

I  think  I  will  use  what  I  learned  about  mood  management  in  other 


situations. 

Strongly 

Di sagree 

Strongly 

Agree  Agree 

Uncertain 

Di sagree 

12%  62% 

17% 

10% 

0% 

Comment:  Seventy  four  percent  of  the  soldiers  agreed  that  the  lesson  on  mood 
management  will  help  them  in  facets  of  their  lives,  outside  of  the  realm  of 
instruction  (see  question  7).  Thus  the  mood  management  lesson  was  perceived  to 
have  more  of  a  general  value  than  in  the  mastering  of  lessons.  The  question  is 
raised.  Is  this  the  principal  intent  of  the  mood  management  lessons? 


Question 

The  lessons  will  help  me  in  my  MOS. 

Strongly  Strongly 

Agree  Agree  Uncertain  Disagree  Disagree 

7*  43*  21*  24*  5* 


Comment:  It  must  be  remembered  that  soldiers  were  not  assigned  to  the  lessons 
based  on  either  their  need  for  skill  remediation  or  because  these  lessons  were 
required  for  their  MOS.  For  example,  how  much  would  cooks  or  pumpers  and 
gagers  perceive  a  need  for  flowcharting,  capitalization,  or  hazard  prevention. 
Additionally,  based  on  pretest  scores  of  lessons,  most  soldiers  were  already  at 
mastery  level  for  most  of  the  lessons. 


Question  10. 

Overall,  the  JSEP  program  was  well  nanaged. 

Strongly  Strongly 

Agree  Agree  Uncertain  Disagree  Disagree 

*  22*  ci  *  c*  n<j 


Comment:  Ninety-three  of  the  soldiers  agreed  the  simulation  was  well 
managed. 


Question  11 . 

I  would  be  willing  to  take  JSEP  lessons  offered  on  a  computer  if 
they  were  offered  during  on-duty  hours. 

Strongly  Strongly 

Agree  Agree  Uncertain  Disagree  Disagree 

38*  48*  14*  0*  0* 


Comment:  Eighty-six  percent  (86%)  agreed  they  would  be  willing  to  take  JSEP 
courses  on-duty.  Thus  soldiers  appear  to  be  very  favorably  disposed  toward 
receiving  the  kind  of  instruction  encountered  in  the  simulation. 
Significantly,  no  one  disagreed  with  the  statement. 


Question 

I  would  be  willing  to  take  wore  JSEP  lessons  offered  on  a  computer 
if  they  were  offered  during  off-duty  hours. 

Strongly  Strongly 

Agree  Agree  Uncertain  Disagree  Disagree 

7%  29%  26%  19%  19% 


Comment:  The  responses  indicate  that  soldiers  are  less  favorably  disposed  to 
taking  JSEP  on  their  own  time.  kevertheless,  only  38%  disagreed  with  the 
statement.  This  outcome  must  be  interpreted  taking  into  consideration  that  many 
of  the  soldiers  saw  no  need  for  the  instruction  since  they  had  already  mastered 
most  of  the  material.  Perhaps  more  would  be  favorably  disposed  to  taking  JSEP 
on  their  own  time  if  they  perceived  a  pressing  need  for  it,  e.g.,  raising  a  GT 
score  to  qualify  for  reelistment  in  the  Army. 


Question  13. 

I  think  my  unit  commander  would  be  willing  to  release  me  from  duty 
to  take  JSEP  lessons. 


Strongly 

Agree 

Agree 

Uncertain 

Di sagree 

Strongly 
Di sagree 

17% 

43% 

36% 

2% 

2% 

Comment:  Sixty  percent  agreed  with  the  statement  and  only  4%  disagreed. 
Consequently,  most  of  these  soldiers  perceived  there  would  be  command  support 
for  such  a  program  and  that  the  program  will  be  worthwhile  for  the  Army.  This 
item  was  also  designed  to  allow  soldiers  to  "project"  on  to  their  commanders 
their  real  feelings  about  the  worth  of  the  experience.  Thus,  this  item  may  be 
one  of  the  more  critical  appraisals  of  the  JSEP  experience. 


Question  14. 

I  think  JSEP  lessons  will  help  me  read  and  understand  the 
publications  I  use. 

Strongly  Strongly 

Agree  Agree  Uncertain  Disagree  Disagree 

24%  57%  17%  2  0 

Comment:  Eighty-one  percent  of  the  soldiers  felt  the  lessons  would  help  them 
to  better  understand  written  materials  encountered  in  the  Army. 


Question  15. 


The  skills  I've  learned  in  JSEP  will  help  ne  advance  to  a  higher 
grade/rank  in  the  Any. 

Strongly  Strongly 

Agree  Agree  Uncertain  Disagree  Disagree 

10t  12%  52 %  17%  OS 


Comments:  Few  (22%)  soldiers  saw  the  potential  for  JSEP  to  help  them  gain 
promotions.  The  reason  for  this  may  be  similar  to  the  possible  explanation  of 
question  12  results. 


Question  16. 

I  think  JSEP  should  be  included  as  an  educational  program  offered 
by  the  Education  Center. 


Strongly 

Agree 

Agree 

Uncertain 

Di sagree 

Strongly 
Di sagree 

48% 

43% 

10% 

0% 

0% 

its  potential  inclusion  in  the  Army's  Educational  offerings, 
percent  agreed  with  the  statement. 


r  ^*1  ^  r\  r* 

3  •  •  -  .  •  ^ 

Ninety-one 


Summary  Conclusions 


1.  Soldiers  had  little  difficulty  learning  to  use  the  computer. 

2.  Four  hours  is  not  too  long  for  a  learning  session  on  the  computer. 

3.  Soldiers  did  not  prefer  instructor-taught  lessons  to  the  computer  based 
lessons. 

4.  The  relationship  between  mood  management  lessons  and  helping  to  learn 
lesson  material  was  not  apparent  to  the  soldiers,  but  they  did  perceive 
the  relevance  of  mood  management  to  other  facets  of  their  lives. 

5.  The  soldiers  saw  the  major  utility  of  the  lessons  in  helping  them  to  read 
and  understand  published  material.  However,  they  perceived  less  utility 
for  MOS  job  performance  and  even  less  for  helping  them  acquire  skills  to 
gain  promotions. 

6.  The  soldiers  very  much  enjoyed  their  experience  during  the  trial  and  would 
like  to  see  JSEP  become  part  of  the  Education  Center's  offerings. 


H.  DEMONSTRATION  YEAR 


The  demonstration  year  is  Phase  II  of  the  JSEP  Implementation  effort.  In 
March  of  1984,  representatives  from  Education  Division,  ODCSPER,  ARI,  and  FSU 
met  to  replan  and  reschedule  JSEP  implementation.  At  the  beginning  of  the 
development  contract,  September  1982,  it  was  envisioned  that  Army-wide 
implementation  would  immediately  follow  curriculum  development.  During  1984  it 
became  apparent  that  the  cyclical  nature  of  contracting  for  learning  center 
staff,  and  the  U.S.  Army  budget  and  procurement  processes  would  provide  a  one 
or  two  year  gap  between  curriculum  development  and  stage  one  of  the  phased 
Army-  wide  implementation.  This  provides  an  excellent  opportunity  to  schedule 
activities  which  will  contribute  to  the  eventual  success  of  that 
implementation.  Three  phases  of  activity  were  planned.  The  first  phase  will 
be  dissemination  of  the  objectives  and  test  questions  for  the  curriculum. 

Phase  II  will  oe  a  year-long  demonstration  at  four  sites.  The  final  phase. 
Phase  III,  will  be  technical  consulting  to  the  actual  implementation  sites. 

All  three  phases  are  concerned  with  transmitting  information. 

The  demonstration  year  will  provide  an  opportunity  for  Army  personnel  to 
see  a  successful  program  in  a  relevant  long-term  setting.  It  will  also  provide 
a  setting  for  advanced  training  of  personnel. 

While  this  effort  is  not  directly  the  concern  of  FSU  under  the  present 
JSEP  contract,  its  existence  has  affected  some  of  the  decisions  concerning  this 
contract,  and  FSU  personnel  are  able  to  offer  recommendations.  One  major 
recommendation  was  that  the  demonstration  sites  be  the  same  as  the  field  trial 
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the  equipment  twice  in  a  short  period  of  time. 


I.  IMPLEMENTATION 

It  is  intended  that  JSEP  have  a  management  system  that  could  assign  other 
ACES  components.  This  assignment  might  be  to  McFann-Gray  materials,  reading, 
English  as  a  second  language,  etc.  The  umbrella  concept  was  the  notion  that 
all  programs  be  related. 


Statement  of  Work  for  JSEP  Coordinator 

The  job  duties  of  the  JSEP  coordinator  in  learning  center  staff  have  been 
defined  in  a  statement  of  work.  CBI  instruction  requires  different 
facilitation  skills  than  one-to-one  or  small  group  tutoring.  It  was  recognized 
that  the  contracting  cycle  of  the  individual  learning  centers  required  advanced 
information  about  the  types  of  skills  required  of  future  staff. 

With  the  time  frames  and  the  nature  of  CBI  in  mind,  FSU  staff  and  persons 
familiar  with  staffing  at  the  CBI  facilities  were  asked  to  contribute  lists  of 
job  duties.  These  lists  were  then  distilled  by  eliminating  duplicates. 

The  SOW  includes  all  the  job  duties  connected  with  JSEP;  it  does  not 
necessarily  represent  the  duties  of  one  person  only.  The  staffing  pattern  and 
numbers  of  staff  will  be  dependent  on  the  number  of  students  per  class,  and  the 
skills  available  in  the  job  market. 
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We  have  carried  out  an  extension  of  this  activity.  One  of  our  staff 
members  has  contacted  the  PLATO  lab  staff  of  the  Army  sites,  interviewing  them 
about  the  conditions  of  their  jobs.  From  these  interviews  we  have  developed 
guidelines  for  instructional  materials  for  staff. 

Exhibit  -  Statement  of  Work 

Present  staff  may  or  may  not  fit  into  the  staffing  profile  for  JSEP. 


STATEMENT  OF  WORK  FOR  JSEP  COORDINATOR 


Background 


1.  Work  experience  in  training  or  education  with  adults. 

2.  Work  experience  in  or  basic  knowledge  of  CBI  or  CAI 
instruction. 

4.  Good  oral  communication  skills. 


Pre-JSEP  Training 

1.  Successful  completion  of  training  in:  JSEP  concepts  and 
formats,  learner  strategies,  terminal  operations, 
PLATO/TICCIT  management. 

2.  Participate  in  periodic  conferences/workshops. 
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Computer  Operation 

1.  Operate  computer  terminals;  use  terminals  to  enter  required 
data;  make  necessary  printouts;  enter  student 
data/information;  create  and  update  permanent  student  record 

^  *  1  *  •  •  A"  i  ^  a  ^ 

I  I  |C  )  I  C.  3  *  .3  ut  1  JwuviCimji 

2.  Demonstrate  system  to  interested  visitors. 

3.  Work  as  trainer  for  new  or  replacement  instructor(s). 


Student  Management 

1.  Receive  students  assigned  to  the  JSEP  program. 

2.  Record  and  report  daily  attendance  of  soldiers. 

3.  Assist  students  in  using  terminals,  and  in  following  JSEP 
procedures. 

4.  Assist  students  in  the  use  of  Handheld  Tutor. 
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5.  Answer  student  questions  concerning  terminal  operations,  JSEP 
content,  policy,  and  logistics. 

6.  Monitor  test  taking  to  assure  test  security. 

7.  Operate  the  CAI  system  in  student  mode: 

a.  Sign  on  new  student 

b.  Enter  relevant  entry  and  test  data 

c.  Detect  student  problems  with  logic  of  instruction  and 
place  student  at  correct  location  in  instruction. 

8.  Administer  tests  and  other  instruction  that  may  accompany 
JSEP. 

9.  Manage  JSEP  students  through  the  use  of  information  provided 
in  the  data  management  system. 

10.  Conduct  student  orientation  to  JSEP. 

11.  Accumulate,  maintain,  and  distribute  any  supplementary 
material  (e.g.,  required  printouts,  charts  for  pacing  module, 
paper  and  pencil). 

12.  Operate  any  equipment  used  in  JSEP  such  as  tape  recorders, 
interactive  videodiscs,  etc. 

J.3.  Prepare  tor  handling  computer  down  time  and  other 
extraordinary  situations. 

14.  Maintain  any  required  records  in  addition  to  those  kept  by 
the  computer  system. 

15.  Assist  students  in  selecting  appropriate  lessons  following 
the  data  received  from  diagnostic  testing  and  instructions 
from  contractor  and/or  Education  Center  Director. 

17.  Provide  accurate  and  motivating  feedback  to  students. 

18.  Document  student  comments  on  the  lessons  and  note  criticisms 
or  problem  areas. 


Computer  Maintenance 

1.  Conduct  basic  troubleshooting  and  maintenance  of  video 
equipment,  cassette  recorders,  videodiscs,  computer 
terminals,  and  other  media  equipment. 


2.  Recognize  system  errors  and  problems,  and  document  these. 
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Instruction 


1. 

2. 


Administer  pencil  and  paper  tests  and  interpret  results 
according  to  lesson  specifications. 

Prescribe  appropriate  lessons  for  student  according  to  test 
results. 
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3.  Direct  students  to  alternate/supplementary  learning  I-S-; 

materials.  Perform  some  basic  library  and  filing  functions  ^ 

in  order  to  maintain  and  track  these  materials.  ili 


4. 

Administer  learning  strategy  lessons  according  to 
specifications. 

5. 

Use  a  variety  of  motivational,  anxiety  reduction,  mood 
management,  and  problem-solving  techniques  (as  introduced  in 
workshops) . 

*  6. 

Teach  lessons  or  skills  which  cannot  be  presented 
evaluated  on  the  computer. 

or 

*  7. 

Use  reinforcement,  self-monitoring,  and  behavioral 
contracts;  counsel  students  on  the  application  of 
techniques  to  individual  situations. 

these 

*  8. 

Use  alternate  instructional  methods,  including  implementing 
policies  concerning  failure  and  the  repetition  of  lessons. 

10. 

Administer  (or  assist)  posttest. 

11. 

Collect  and  interpret  post-cycle  data  and  provide  Education 
Center  requested  and/or  contractor  requested  records. 
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Hh  end  wllhout  whole  numbers 


Use  an  1 1  lustration  or 
Mquance  of  It  lustrations 
to  follow  directions. 
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